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ABSTRACT 

Ohio*^ Statevide Needs Assessment is designed 
provide baseline datajabout students' learning in cognitive areas^^ 
Over several years, students at various grade levels will be 
assessed.' The areas to be assessed, as well as the grades of the 
students to be assessed, will be considered and advised upon by the 
ESEA Title III State Advisory Council. The council recommended that 
for the first year of assessment, only reading be assessed at grade , 
tvelve. The reading assessment instrument to be used in this first 
year contains items that assess those objectives thought by reading 
specialists in the Ohio Department of Education to be *.he most 
important. It will be administered to only a sample of Ohio twelfth 
graders by school personnel. Confidence intervals for each item on 
the instrument will be calculated in order to predict student 
performance. After the data from the assessment have been analyzed, 
reports presenting and explaining the result^, will be given. Regional 
meetings may be held throughout the state to provide further 
explanation of assessment results, while interpretations and 
implications of Title III are discussed. (Author/BJG) 
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SECTION I ^ 



INTRODUCTION 



.Are students learning what they should be learning? If not, in what 
areas do student learning needs exist? In what areas are student learning 
needs the greatest? 

The answers to these questions are of obvious interest to all concern- 
ed with the educational process - students, teachers, parents, adminis- 
trators, and general citizenry. Ohio's Statewide Needs Assessment, pres- 
ently being developed by the Ohio Department of Education for use in 
the Elementary and Secondary Education Act Title III effort, will 
supply eventual answers to the above questions by providing data about 
the learning needs of Ohio students. 

Information about statewide student learning needs is vital to the 
ESEA Title 1 1 1 effort. The purpose of Title III is to fund innovative and 
exemplary projects which address themselves to alleviating statewide 
learner needs. Those districts applying for Title III funding must show 
evidence that they have collected and analyzed local data which indicate 
the existence of a district learning need. In other words, the district 
must conduct a needs assessment. The district also must demonstrate 
that this local need corresponds to a statewide critical learner need. 

Throughout the years of Title III operation in Ohio, statements 
about statewide student needs have been generated from such sources 
as the Si^ate Board of Education's goal statements and the "Top Citi- 
zens' Conceriis^emanating from the state's massive involvement of citi- 
zens in education — Search for Consensus. 

In 1973, tlie U.S. Office of Education, which annually reviews each 
state's plan for administering Title III, concluded that Ohio's past meth- 
od for determining statewide critical needs should be extended to a 
direct assessment of students. To respond to this dictate from the 
Office of Education, the Division of Planning and Evaluation of the 
Ohio Department of Education has been given the responsibility for 
planning and developing an assessment system that will indicate more 
accurately and specifically what are the statewide student learning 
needs. 



Because Ohio has never before conducted a statewide assessment 
similar to that now being planned, the first assessment, to be conducted 
in October of 1974, is being regarded as a pilot study. This does not 
mean that valuable information will not result. It does mean, though, 
that during this first year, emphasis is being placed upon developing an 
effective, workable assessment procedure.. 



SECTION II 



OHIO'S STATEWIDE NEEDS ASSESSMENT- 
, WHAT IT IS AND IS NOT 

Before elaborating on the purpose of the assessnnent program, it may 
be helpful to begin by stating what the program \snot intended to do. 

First, the program is not one that will involve all schools or all stu- 
dents who are at the age or grade level that has been selected to be 
assessed. Rather, a sample of students from a sample of schools will be 
drawn.: The exact procedure that will be used to select schools and stu- 
dent<i will be explained in this publication. 

Second, Ohio's Statewide Needs Assessment is not a mandatory pro- 
gram. Of course, it is hoped that schools selected to participate will do 
so to assure the representativeness of the sample aHd to permit the re- 
sults to be generalized.. 

Third, the data resulting from the assessment will not be analyzed or 
reported on for individual students, schools, or districts. Indeed, at the 
time of the assessment, names of students will not be collected, nor 
will names of schools or districts be noted on the students' answer 
sheets. 

Fourth, during the first year of the assessment program no student 
socioeconomic data, except age and sex of student, will be collected. 
With the exception of the size of the senior class of each school partici- 
pating in the program, no school, district, or community data will be 
requested. 

Now that an explanation of what Ohio's Statewide Needs Assess- 
ment will not involve has been given, the program as it is being planned 
will be explained. 

Ohio's Statewide Needs Assessment is designed to provide baseline 
data about students' learning in cognitive and, perhaps in future years, 
non-cognitive areas. Over several years, students at various age/grade 
levels will be assessed. The area(s) to be assessed each year, as well as 
the age(s) or grade(s) of the students to be assessed, will be considered 
and advised upon by the ESEA Title III State Advisory Council.; Mem- 
bers of the advisory council, representing educational, cultural, and 
business interests throughout the state, are appointed to the council by 
the Superintendent of Public Instruction. 
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In the fall of 1973, upon the advice of the advisory council, it was 
decided that, for the first year of the assessment, only one area-read- 
ing-will be assessed at only one grade level-the twelfth. Twelfth grad- 
ers were selected because they represent the terminal point in stuudnts' 
seconaary education. 

The decision to collect data about student reading reflects one of the 
concerns expressed by citizens during the state's Search for Consensus. 
This concern and others were adopted as statewide educational goals in 
June, 1973, by the State Board of Education. These goals, which foU 
low, will serve as guides when future decisions are made about the areas 
to be assessed. 



Statewide Educational Goals 
BASIC ACADEMIC SKILLS 

Language Skills 

Each student should h<jve hasie reading, writing, hstening and speak- 
ing skills. 

Mathematics and Science v 

Each student should luive baste analytical and reasoning skills in- 
cludnig a knowledge of mathematics and science facts 

Social Studies 

' Each student should know the l)asjc principles and practices of 
American society in comparison to other societies, as well as the 
rights and duties associated with responsible citizenship in American 
society 

Capabilities of Aesthetic Experience 

Each student should have the opportunity to learn the fundamen- 
tals of,^ to observe and participate in music and the other fine and 
performing arts. 

Career Education 

Each student sbo4.ild have the skills necessary to be able to make a 
career decision and be qudlified for a career choice 

Human Relations, Family Li ving and Personal De v elopment 

Each student should have an understanding of the importance of 
human relations, family relationships and the skills necessary to 
function successfully in everyday life. 
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Learning To Be a Learner 

Each student should know how to obtain and use new facts, skills 
ideas as his need tor them arises. 

Physical Fitness, Recreation antlMentat Heatth 

Each student should know how to develop and maintain acceptable 
physical and mental health. 

The same age or grade level of students will be reassessed periodically 
in the same area(s), thus allowing for a measurement of student 
achievement over several years. For example, the first year of the 
assessment program now being planned will assess reading among 
twelfth graders. Perhaps five years from now the reading ability of 
twelfth grade students again may be assessed. The results of both of 
these assessments could be compared to make some determination as 
to whether or not reading performance of Ohio twelfth graders has im- 
proved, remained the same, or worsened in that time. 

These comparisons of similar groups of students will be particularly 
valuable because of the approach to assessment that will be utilized in 
t^-program. Ohio's Statewide Needs Assessment will employ an ob- 
jective-based or criterion-referenced assessment instrument rather than 
a norm-referenced instrument. 

The latter is designed to result in a ranking of students in order from 
high to lovj/, depending upon how well they performed on the test 
instrument. Since the intent of Ohio's Statewide Needs Assessment is 

n objective- 



not to compare individual students, schools, or districts,^ 
based instrument was thought to be more suitable than a norm-refer 
enced instrument.; 
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An objective-based assessment instrument assesses the student's 
ability to perform tasks thought to be desirable for the student to per- 
form. These desired student performances or tasks are e>cpressed in 
terms of objectives. Therefore, an objective is a statement of adesfjf^d 
studeoCp^rformance. The objective based instrument contains items 
desfgried^to determine whether or not students are able to do what the 
objectives state they should be able to do. A student's performance on 
the instrument then is judged m terms of how well he or she can per- 
form the objectives rather than how well he or she performs in relation- 
ship to other students. 
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SECTION III 



HOW WAS THE ASSESSMENT INSTRUMENT DEVELOPED? 



Once the decisions about grade level and subject area to be assessed 
during year one were nnade, the Division of Planning and Evaluation 
collected existing objectives and test itenns suitable for usage in the 
first year of the needs assessnnent* Fronf>the onset of the assessment 
planning, the division was interested m surveying the field of existing 
objectives and itenns. The .writing of good objectives and itenns is tinne 
consuming and costly. Therefore, the first step was to determine what 
objectives and items existed before undertaking the development of 
new ones. 



The major source of the existing objectives and test exercises was 
NAEP - National Assessment of Educational Progress - a project 
sponsored by the Education Commission of the States. NAEP assesses 
young Americans at the ages of nine, 13, t7 and 26-35 in a number of 
subject areas. Each subject area is assessed on a rotating basis. In other 
words, an area is assessed every few years rather than evei^y year. After 
4ach assessment is conducted, NAEP releases the objectives and some 
of the items used. These items become a part of the public domain, 
which means that they can be used by others at no cost. \ 

In the collection of the objectives and released items that, NAEP used 
in its assessment of reading, particular attention waX given to those ob- 
jectives and items used to assess 17-year-olds. \ 

Another source of existing objectives and items was other states. 
When the division began planning for the arsessment, it cor;^ducted a 
survey of other states' assessment programs. This survey indicated, 
among other things, what subject areas and age/grade levels of students 
were assessed by other states, and which states were utilizing an opjec- 
tive-based approach. Consequently, the division was able to identify 
those states planning to assess 17-year-olds or twelfth^ graders in the 
area of reading using an objective-based instrbment. 

Requests were sent to these states for examples of their objectives 
and items. Virtually all of these states' assessment were composed 
either totally or in part of NAEP objectjves and items. Some states had 
undertaken to develop their own objectives and items to supplement 
NAEP materials but had not yet conducted their assessments. Others 
had, but were not yet ready to release their materials. One state— 
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Minnesota-did grant the department permission io include some of its 
objectives and itenrrs in Ohio's assessment of twelfth grade reading. 

After the, existing objectives had been collected, reading specialists 
from the Qhio Department of Education reviewed them to determine 
which objectives would be most important anjd suitable to include in 
the first assessment of reading among twelfth graders. 

* 

Items existed to assess some of these objectives. For other objectives, 
however, few or no suitable items existed. For these objectives, new 
items were developed. The division began this procedure by writing a 
Request For a Proposal (RFP) for item development. The RFP listed 
those features the division wanted the new items to include. 

This request was sent to several companies identified as experienced 
in objective-based item developrvient. The proposal submitted by the 
Measurement Research Cfinter-^iuision of Westinghouse Learning Cor- 
poration, was judged to be ihe most suitabl: and the contract for de- 
velopment of new test items for year one of the assessment was granted 
to Measurement Research Center. 



An initi'al part of the development of the assessment instrument was 
the inrvolvement of a review panel composed of ^administrators, teachers, 
students, and representatives 'df educational ahd civic organizations. 
The function of this panel, which met on March *8, 1974, was to deter- 
mine whether or not the objectives thought by the reading specialists to 
be most important to assess during year one of the program, and the 
itenns for these objectives, met the objective and 'tem'criteria included in 
the ESE A Title 1 1 1 State Plan. Specifically, the department wanted each 
merrber of the review panel to determine whether or not each objec- 
tive represented an impoctant part of a student's learning or skill in 
reading; learning that would be appropriate to assess among twelfth 
graders; learning that is taught in Ohio schools; and learning that should 
be taught in Ohio schools. 

In the opinion of the review panel members, most of the objectives 
were important, appropriate, and should be taught. The criterion the 
objectives most frequently failed to meet was the is taught in Ohio 
schools criterion. Nevertheless, nf\pst objectives which did not meet this 
criterion were deemed important ehough to be included. 

The panel members were asked to apply a second set of criteria to 
the items. First, they read each item to assure that it was not offensive, 
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Obviously, the assessment instrument was not to include items that 
utilized language or examples that could constitute an invasion of pri- 
vacy or that could be construed as offensive. 

Second, the items were reviewed for relevancy. As much as possible, 
items were to present students with reading situations similar to those 
which they would encounter in their everyday adult activities. 

Third, items were to be underst3ndable. In other words, the items 
were to be clearly written and were to communicate their intent. 

Foun\{, the rpview panel determined whethe** or not the item had 
directionalrty. fiach item was to have only ono correct or desire'^ 
answer. 

Finally, the panel members were to determine whether each item 
was a direct rheasure of the objective it was intended to assess. Od- 
viQusly, each item was to measure thejtudent's ability to perform the 
objectTve it was designed to assess. 

The -"eview panel members thought few of the items were offensive, 
irrelevant, unclear, or not a measure of their objective. The criterion 
the items most frequently failed to meet was that of directionality. 

Based upon the reviews and comments of the panel members, some 
items were altered and corrected; others were excluded from further 
consideration. 

In summary, the reading assessment instrument to be used in the 
first year of Ohio's Statewide Needs Assessment will contain items that 
assess those objectives thought by reading specialists in the Ohio De- 
partment of Fducation to be most important to include in the first 
assessment of reading among twelfth graders. The items contained in 
the instrument come from three sources — the National Assessment of 
Educational Progress, the State of Minnesota, and the Westinghouse 
Learning Corporation. 
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SECTION IV 

WHO WILL BE INCLUDED IN THE ASSESSMENT? 



Ohio's Statewide Needs Assessment is planned to assess only a sam- 
ple of Ohio twelfth graders. Dr. D. Ransom Whitney, Chairman of the 
Department of Statistics, Ohio State University/ anc^ the Ohio Depart- 
ment of Ed»!catiori Division of Computer Services, have aided the de- 
partment ' ■ "^Inu the assessment sample 

A two .ia^ed sampling approach is to be utilized. First, a random 
sample of schools was drawn from all Ohio public and nonpublic 
schools that have a twelfth grade class. Superintendents from the dis- 
tricts in which these schools are located will be contacted and asked to 
participate in the assessment. It should be repeated here that participa- 
tion in Ohio's Statewide Needs Assessment is voluntary. However, to 
assure representativeness of the sample and the ability to generalize the 
results, the department hopes all sample schools will dedide to take 
part in the assessment. ; 

Personnel from schools agreeing to participate will conduct the 
seconc^ stage of the sampling procedure - i.e., the selection of the ran- 
dom sample of students from the schools' twelfth grade clashes. The 
exact number of students involved in the assessment from each school 
will depend upon the size of the twelfth grade class in that school. The 
procedure that school personnel will use to draw the student sample 
will be developed by the department and explained at a workshop 
planned for September, 1974, It 'S important to stress that the actual 
selection of the student sample wni made by personnel from the 
school, not from the department. Thus, the names of students partici- 
pating in the program will not at this time or at any other time be 
known by the department. ' ' 
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SECTION V 

HOW WILL THE INSTRUMENT BE ADMINISTERED? 



It is apparent that the administration of the instrument can best be 
accomplished I by school personnel - preferably guidance counselors. 
The counsel(^rs or personnel selected "from the schools to administer 
the instrument will attend a workshop probably in September, 1974, 
at which time the exact method to be used to administer the instru- 
ment will be explained. The schools will be asked to conduct the assess- 
ment on a day during the fourth week in October. 

Most students should require about one and one-half hours to com- 
plete the test section of the instrument. Additional time will be allotted 
for the test administrator to give directions and for students to com- 
plete preliminary questions about their attitudes toward reading. Too, 
students wtll be given about a five to ten minute br^ak midway in the 
assessment. 

It is estimated that most students should be able to complete the 
preliminaries and the test section in two hours.; However, because the 
intent of the assessment program is to give students the opportunity to 
answer all the items in the instrument, the schools will be asked to pro- 
vide enough time to allow the vast majority of students to complete all 
the items. 

The department will request that all students in the sample within a 
school be assessed at the same time. The number of students to be 
assessed in most schools involved will be less than 30. If schools are 
unable to assess all students at one time, special arrangements can be 
made for a split assessment session. 

The school personnel who administer the instrument will be respon- 
sible for mailing the test booklets and answer sheets to the Department 
of Education. The personnel also will be responsible for informing the 
department as to the exact number of twelfth grade students in the 
school at the time of the assessment (October, 1974). Why the depart- 
ment needs this information is explained 'n the answer to the next 
question. 



ERIC 



15 



I SECTION VI 

HOW WILL ASSESSMENT DATA BE ANALYZED AND REPOR-^ED? 
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As the answer sheets' are returned to the department, a notation will 
be made on each answer sheet to indicate the size of the twelfth grade 
class of the student who completed the answer sheet. This information 
is needed by the sampling consultant to calculate the confidence inter- 
val for each item. (An explanation of confidence interval will be given 
later.) To supply this information the department will employ a coding 
system that will allow the sampling consultant to have the data to 
make the calculations, while maintaining the anonymity of the schools. 
Categories based upon the twelfth grade class size of schools involved 
in the assessment will be established. A code letter (A,B,C) will be 
assigned to each category. For example, category A schools may be 
those with a twelfth grade class size between 0-100 students. Category B 
schools may be those with a class size between IOt-200, and so on. The 
answer sheets of those students who attend schools with a twelfth 
grade class size betwee.. 0 and 100 will be coded wijLh a letter A, while 
students attending schools with a twelfth grade class size between 101 
and 200 will be coded with a B, and so on. 

Because no category will contain only one school, after this code is 
placedToTi lhe studerits^ answer sheets it will be imppssibje to identify 
the school at which any answer sheet was completed. It should be 
stressed once again that this categorization is being done only to calcu- 
late the item confidence interval. These categories will not be used for 
further analysis or for reporting the results.. 

The reader may be asking what is a confidence interval and why is 
its calculation necessary? A confidence interval is the range of scores 
within which it can confidently be assumed that the actual score falls. 
An actual score used in this sense m^ans the score that could be expect- 
ed if all students in the group to be assessed were involved in the assess- 
ment rather than just a sample of these students. 

When the confidence intervals for each item on the reading assess- 
ment instrument are calculated, the performance of all Ohio twelfth 
grade students on each item in the instrument can be predicted from 
the performance on the items by a sample of Ohio twelfth grade stu- 
dents. 
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The assessment instrument will be scored so that the percentage of 
' students that correctly answered each itenn as well as the percentage of 
\ students that selected each of the incorrect answer choices will be 
known. 

Knowing how students in the assessnnent responded on each item 
and knowing the confidence interval of each of the items, those inter- 
preting the results may make statements such as the following: tf on 
item one of the assessment 75 percent of the students>esponded cor- 
rectly and if the confidence interval for that item has been calculated 
as plus or minus two percent, it can be said that if all Ohio twelfth 
graders were involved in the assessment, chances are good that between 
73 percent and 77 percent of the students would answer the item cor- 
rectly. The reader might ask, "Just how good are the chances that the 
percentage of Ohio students responding correctly to item one would 
fall into that range?" Year one of Ohio's Statewide Needs Assessment 
will yield results at the 95 percent confidence level. This means that 
if 75 percent of the students responded correctly to item one,and a plus 
or minus two percent confidence interval has been calculated, the 
chances are 95 out of 100 that the actual score of all Ohio students on 
item one falls between 73 percent and 77 percent. Therefore, theoreti- 
cally a five percent chance exists that the percentage of students cor- 
rectly responding to the item would fall below 73 percent or above 77 
percent. 

These item-by-item analyses will give a good indication of how well 
students can perform the objectives being assessed.. However, to deter- 
nine where student needs exist it must be determined how well stu- 
dents should be able to perform the reading objectives^ In other words, 
desired student performance levels must be established for each item. 
A procedure to establish these desired performance levels is now being 
planned. It is anticipated that teachers, administrators, representatives 
of educational and civic organizations, and students will be involved in 
this process. These individuals will be asked to look at each item in the 
assessment instrument and ask themselves, "What percentage of twelfth 
grade students should be able to answer this item correctly?" 

Knowing what students should be able to do and what they actually 
did will point out where student reading needs exist with respect to the 
objectives assessed. 

The following illustration will explain how this will be accomplished. 
Returning to the example used earlier, if 75 percent of the students re- 
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sponded correctly to item one, but those setting item desired perfor- 
mance levels thought 90 percent of the students should be able to re- 
spond correctly to the item, it could be said that a 15 percent discrep- 
ancy exists between what Ohio students can do on the item and what 
it is thought the student should be able to do. If these calculations are 
made for each item, a determination can be made about student needs 
for each item* If all items that assess a given objective are grouped to- 
gether, a determination of student needs for that objective can be made. 

The data from the assessment may be analyzed in other ways. In 
what ways will be decided follov,/ing discussions with people experi- 
enced in analyzing data from states' assessment programs* These dis- 
cussions will be held in approaching months. It is anticipated that by 
early 1975 the results of the data analysis should be known* 



\ SECTION VII 

HOW WILL TH€ RESULTS BE DISSEMINATED? 



After the data from the assessment have been analyzed, reports pre- 
senting and explaining the results will be written. It also is anticipated 
that regional meetings will be held throughout the state to provide fur- 
ther explanation of the assessment results to reading specialists, 
teachers, school administrators, and other interested people. Emphasis 
at these meetings will be upon the discussion of the interpretations a id 
implications of the results for Title 1 1 1 purposes. 
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SECTION VIII 

HOW WILL SUBSEQUENT YEARS OF 
THE ASSESSMENT BE CONDUCTED? 



When the first year of the assessment is completed, planning will be- 
gin for the second year of the program. Anajysis of the first year activi- 
ties will be made to determine what similar activities need be under- 
taken in subsequent years of the assessment and what activities need be 
altered. As the program progresses over the years, alterations will be 
made where and when they are needed. 



SECTION iX 

IN CONCLUSION 



As Ohio's Statewide Needs Assessment becomes an established fea- 
ture of the Title III effort, the learning needs of students at various 
grade levels in several cognitive and noncognitive areas will be defined 
accurately and specifically. Those vying for Title III funding will be ex- 
pected to address themselves to these needs and attempt to design pro- 
jects to alleviate them. 
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